Control apparatus, control method, and storage medium

ABSTRACT

A control server includes an information acquisition unit configured to acquire information indicating an imaging condition of a plurality of imaging apparatuses, a communication parameter determination unit configured to determine a time length during which each of a plurality of pieces of image data based on image capturing by the plurality of imaging apparatuses is communicable, based on the information acquired by the acquisition unit, and a control unit configured to perform control so that communication of the plurality of pieces of image data is performed in accordance with the time length determined by the determination unit.

BACKGROUND Field

The present disclosure relates to control of data communication.

Description of the Related Art

Attention has recently been being given to a technique for generating animage (virtual viewpoint image) viewed from a specified viewpoint(virtual viewpoint) by using a plurality of images captured by aplurality of imaging apparatuses disposed around an imaging region. Thetechnique for generating a virtual viewpoint image enables users toview, for example, highlight scenes of soccer and basketball fromvarious angles, thus providing users with a high degree of realisticsensation in comparison with regular images.

Japanese Patent Application Laid-Open No. 2017-211828 discloses imagedata communication performed by a plurality of image processingapparatuses. Each of the image processing apparatuses discussed inJapanese Patent Application Laid-Open No. 2017-211828 extracts a regioncorresponding to an object (hereinafter referred to as an object region)from a captured image acquired by an imaging apparatus connected to theimage processing apparatus. The image processing apparatus transmitsimage data representing the extracted region to an apparatus forgenerating a virtual viewpoint image, via other image processingapparatuses connected in a daisy chain.

A description will be provided of consideration about a plurality ofcaptured images acquired by a plurality of imaging apparatuses includingimaging apparatuses having different imaging conditions. For example, ina case where the plurality of imaging apparatuses includes imagingapparatuses having different focal lengths as an imaging condition, anobject captured by an imaging apparatus having a longer focal length iscaptured in a zoom way in comparison with an object captured by animaging apparatus having a shorter focal length. Thus, it is assumedthat the object region in the captured image acquired by the imagingapparatus having a longer focal length is larger than the object regionin the captured image acquired by the imaging apparatus having a shorterfocal length. The size of the object region may differ from each imagingapparatus depending on imaging conditions of the imaging apparatus, suchas the distance between the imaging apparatus and the imaging region inaddition to the focal length described above.

As the size of the object region increases, the data amount of imagedata representing the object region increases. Accordingly, the timeperiod required to transmit the image data may also increase inaccordance with the data amount. In this case, when a plurality ofpieces of image data having different data amounts is transmitted, imagedata having a large amount cannot be transmitted within a predeterminedtime duration depending on, for example, communication limitations orthe receiving capability of the reception side, possibly resulting inmissing image data. In a system including a plurality of imagingapparatuses, image data transmission may not be suitably performed asdescribed above depending on the imaging conditions for each imagingapparatus.

The present disclosure has been devised in view of the above-describedissues. The present disclosure is directed to preventing image data frombeing lost due to the increase in the data amount of image datadepending on imaging conditions.

SUMMARY

A control apparatus includes an acquisition unit configured to acquireinformation about an imaging condition of a plurality of imagingapparatuses, a determination unit configured to determine a time lengthduring which each of a plurality of pieces of image data based on imagecapturing by the plurality of imaging apparatuses is communicable, basedon the information acquired by the acquisition unit, and a control unitconfigured to perform control so that communication of the plurality ofpieces of image data is performed in accordance with the time lengthdetermined by the determination unit.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a configuration of an image processing system.

FIG. 2 illustrates a function configuration of a camera adapter.

FIG. 3 illustrates examples of a captured image and a foreground image.

FIG. 4 illustrates other examples of a captured image and a foregroundimage.

FIG. 5 illustrates a function configuration of a control server.

FIGS. 6A and 6B are timing charts illustrating timing of image datatransmission in communication based on a time-division multiplex method.

FIGS. 7A and 7B are timing charts each illustrating timing of image datatransmission in communication based on a bandwidth multiplex method.

FIGS. 8A and 8B are timing charts each illustrating timing of image datatransmission in communication based on a token passing method.

FIG. 9 illustrates examples of camera arrangements.

FIG. 10 illustrates examples of captured images acquired in imagecapturing by the cameras.

FIG. 11 is a flowchart illustrating processing performed by the cameraadapters and the control server.

FIGS. 12A and 12B are timing charts illustrating a method in which aplurality of cameras including cameras having different imaging framerates transmits image data based on the time-division multiplex method.

FIG. 13 illustrates a hardware configuration of the control server inthe image processing system.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present disclosure will be described belowwith reference to the accompanying drawings. Components according to thefollowing exemplary embodiments are to be considered as examples ofembodiments of the present disclosure, and the present disclosure is notlimited to these exemplary embodiments.

A first exemplary embodiment of the present disclosure will be describedbelow. The present exemplary embodiment will be described belowcentering on an image processing system for generating a virtualviewpoint image. A virtual viewpoint image refers to an imagerepresenting a view from a specified viewpoint based on a plurality ofimages captured by a plurality of imaging apparatuses and a specifiedarbitrary viewpoint (virtual viewpoint). A virtual viewpoint imageaccording to the present exemplary embodiment is also referred to as afree viewpoint image, and is not limited to an image corresponding to aviewpoint freely (arbitrarily) specified by the user. For example, animage corresponding to a viewpoint selected from among a plurality ofcandidates by the user is also included in virtual viewpoint images.Virtual viewpoint images according to the present exemplary embodimentmay be static or moving images. The image data handled by the imageprocessing system may be static or moving images. In other words, theimage processing system according to the present exemplary embodimentcan process static and moving images.

A hardware configuration of each apparatus included in the imageprocessing system according to the present exemplary embodiment will bedescribed below with reference to FIG. 13. While the hardwareconfiguration of a control server (described below) will be describedbelow as an example, camera adapters and an image processing server(described below) may have a similar hardware configuration. A controlserver 300 illustrated in FIG. 13 includes a central processing unit(CPU) 1401, a read only memory (ROM) 1402, a random access memory (RAM)1403, an auxiliary storage device 1404, a display unit 1405, anoperation unit 1406, a communication interface (I/F) 1407, and a bus1408.

The CPU 1401 controls the entire control server 300 by using computerprograms and data stored in the ROM 1402 and the RAM 1403 to implementthe function of each processing unit included in the control server 300.The control server 300 may include one or a plurality of dedicatedhardware components different from the CPU 1401, and at least part ofprocessing of the CPU 1401 may be performed by the dedicated hardwarecomponents. Examples of dedicated hardware components include anApplication Specific Integrated Circuit (ASIC), a Field ProgrammableGate Array (FPGA), and a Digital Signal Processor (DSP). The ROM 1402stores programs that do not need to be modified. The RAM 1403temporarily stores programs and data supplied from the auxiliary storagedevice 1404 and data supplied from the outside via the communication I/F1407. The auxiliary storage device 1404 includes, for example, a harddisk drive and stores various types of data, such as image data andaudio data.

The display unit 1405 includes, for example, a liquid crystal display orlight emitting diodes (LEDs) and displays a graphical user interface(GUI) for the user to operate the camera adapters. The operation unit1406 includes, for example, a keyboard, a mouse, a joystick, and a touchpanel. In response to receiving user operations, the operation unit 1406inputs various instructions to the CPU 1401. The CPU 1401 operates as adisplay control unit for controlling the display unit 1405 and as anoperation control unit for controlling the operation unit 1406.

The communication I/F 1407 is used to communicate with an apparatusoutside the control server 300. For example, in a case where the controlserver 300 is connected via a wire to an external apparatus, acommunication cable is connected to the communication I/F 1407. In acase where the control server 300 has a function of wirelesslycommunicating with an external apparatus, the communication I/F 1407includes an antenna. The bus 1408 connects the units of the controlserver 300 and transmits information.

Although, in the present exemplary embodiment, the control server 300includes therein the display unit 1405 and the operation unit 1406, atleast either one of the display unit 1405 and the operation unit 706 mayexist as different apparatuses outside the control server 300. Eitherone or both of the display unit 1405 and the operation unit 1406 may beabsent. This also applies to the camera adapters and the imageprocessing server (described below).

FIG. 1 illustrates a configuration of an image processing systemaccording to the present exemplary embodiment. An image processingsystem 10 includes cameras 101 to 104, camera adapters 111 to 114, animage processing server 100, and a control server 300. Each apparatuswill be described below.

The cameras 101 to 104 are, for example, digital video cameras and otherimaging apparatuses. The cameras 101 to 104 capture images of theimaging region to acquire captured images and transmit the acquiredcaptured images to the connected camera adapters 111 to 114,respectively. The present exemplary embodiment will be described belowon the premise that the imaging region is a stadium where a sports gameis held. The imaging region can be implemented not only in a stadium butalso in a field, gymnasium, and studio. The cameras 101 to 104 captureimages in a state where the respective imaging timing is synchronized.The cameras 101 to 104 include lenses 121 to 124, respectively. Thelenses 121 to 124 can be wide angle lenses, zoom lenses, or fixed focallength lenses. The focal lengths of wide angle lenses and zoom lensescan be adjusted by the control server 300, the image processing server100, and the camera adapters 111 to 114 (described below).

While in the present exemplary embodiment, the angle of field isadjusted by using lens-based optical zooming, digital zooming may alsobe performed by the cameras 101 to 104 and the camera adapters 111 to114. FIG. 1 illustrates the lenses 121 to 124 so that the differences inthe focal length can be identified, and illustrates that the focallength of the lens 123 is longer than the focal lengths of other lenses121, 122, and 124. Unless particularly distinguished in the followingdescriptions, the lenses 121, 122, 123, and 124 are also collectivelyreferred to as a lens 121.

Effects of using the lenses 121 to 124 having different focal lengthsare described below. In a region where important scenes are likely tooccur (e.g., in front of goals in a soccer game), the use of a telephotolens is desirable since a demand for high-resolution image capturing isassumed. For an imaging apparatus that captures the entire field, theuse of a wide angle lens is desirable. For example, there is assumed asituation where image capturing is to be performed with zooming at atiming when an important scene occurs. In this case, a captured imagecloser to the image demanded by the user can be acquired if the focallength is dynamically adjustable. A plurality of imaging apparatuses areinstalled assuming the above-described situation.

Although, in the present exemplary embodiment, an imaging apparatus inwhich one lens is attached to each camera is used as an example, diversetypes of apparatuses can be used as imaging apparatuses included in theimage processing system 10. For example, an imaging apparatus in which aplurality of lenses is attached to one housing may be used. Although, inthe exemplary embodiment, four cameras are installed, the number ofinstalled cameras is not limited thereto but four or more imagingapparatuses may be installed. Camera arrangements are not limited to theexample illustrated in FIG. 1. For example, the plurality of cameras isset to surround the stadium. Each camera may be an apparatus thatcaptures not only images but also audio and other sensor information.Unless particularly distinguished in the following descriptions, thecameras 101, 102, 103, and 104 are also collectively referred to as acamera 101.

The camera adapters 111 to 114 acquire captured images acquired in imagecapturing by the cameras 101 to 104, respectively, and perform imageprocessing. The image processing will be described in detail below. Thecamera adapters 111 to 114 transfer image data acquired through theimage processing to the image processing server 100. Although, in thepresent exemplary embodiment, the four camera adapters 111 to 114 are tobe used in conformance with the number of cameras 101 to 104, the numberof camera adapters is not limited thereto. For example, a plurality ofcameras may be connected to one camera adapter. Although, in the presentexemplary embodiment, the camera adapters and the cameras are describedas different apparatuses, an imaging apparatus may include the camerasand the camera adapters. Unless particularly distinguished in thefollowing descriptions, the camera adapters 111, 112, 113, and 114 arealso collectively referred to as a camera adapter 111.

The image processing server 100 acquires a plurality of image datapieces transmitted from the camera adapters 111 to 114 and generates avirtual viewpoint image based on the plurality of acquired image datapieces. The image processing server 100 according to the presentexemplary embodiment generates three-dimensional geometric datarepresenting the three-dimensional shape of an object based on theplurality of acquired image data pieces. The image processing server 100generates a virtual viewpoint image by using the generatedthree-dimensional geometric data and information indicating the virtualviewpoint acquired by a user specification. The method for generating avirtual viewpoint image is not limited thereto. For example, a methodfor generating a virtual viewpoint image based on image-based renderingis also applicable. The control server 300 is a control apparatus thatcontrols the image processing system 10. The control server 300 controlsthe cameras 101 to 104, the lenses 121 to 124, the camera adapters 111to 114, and the image processing server 100.

The above-described apparatuses included in the image processing system10 are connected to each other via a local area network (LAN). Thenetwork topology according to the present exemplary embodiment is basedon the daisy chain connection. More specifically, as illustrated in FIG.1, the plurality of camera adapters 111 to 114 is connected in a daisychain. According to the present exemplary embodiment, the camera adapter111 transmits image data acquired through the image processing performedby the camera adapter 111 itself, to the camera adapter 112. The cameraadapter 112 transmits the image data acquired from the camera adapter111 and image data acquired through the image processing performed bythe camera adapter 112 itself, to the camera adapter 113. The cameraadapter 114 at the end of the daisy chain connection transmits the imagedata acquired from the camera adapter 113 and image data acquiredthrough the image processing performed by the camera adapter 114 itself,to the image processing server 100. In the daisy chain connection, theside of the image processing server 100 is the downstream side. Each ofthe camera adapters 111 to 114 is configured to transmit the image datareceived from the corresponding upstream one of the camera adapters 111to 114 and image data acquired by itself, to the correspondingdownstream one of the camera adapters 111 to 114.

A function configuration of the camera adapter 111 will be describedbelow with reference to FIG. 2, as a representative example of thefunction configuration of the camera adapters 111 to 114. The cameraadapter 111 includes an image acquisition unit 400, a foregroundextraction unit 401, a camera information acquisition unit 402, acommunication parameter acquisition unit 403, a data amount reductionunit 404, and a communication unit 405. Referring to FIG. 2, the cameraadapter 111 is directly connected to the control server 300 and theimage processing server 100. However, since the plurality of cameraadapters 111 to 114 is connected in a daisy chain, one that is notdirectly connected to the control server 300 and the image processingserver 100 is present. One, out of the camera adapters 111 to 114, thatis not directly connected to the control server 300 and the imageprocessing server 100 can communicate with the control server 300 andthe image processing server 100 via the corresponding downstream one ofthe camera adapters 111 to 114. Processing units of the camera adapter111 will be described below.

The image acquisition unit 400 acquires an image acquired by the camera101 capturing an image of the imaging region. Here, an image 253illustrated in FIG. 4 is a captured image acquired through imagecapturing performed by the camera 101 in FIG. 1. An image 250 in FIG. 3is a captured image acquired through image capturing performed by thecamera 103 in FIG. 1. The cameras 103 and 101 capture an image of thesame player 202. Since the camera 103 has a longer focal length than thecamera 101, the image 250 has a larger region corresponding to theobject in the captured image than that in the image 253. According tothe present exemplary embodiment, moving objects, such as persons and aball, in the imaging region are referred to as a foreground. In acaptured image acquired by capturing an image of the foreground, theregion corresponding to the foreground is referred to as a foregroundregion. In the imaging region, objects different from the foreground isreferred to as a background.

The foreground extraction unit 401 extracts the foreground region fromthe captured image acquired from the camera 101 by the image acquisitionunit 400, and performs image processing of generating image data(hereinafter referred to as a foreground image) including the extractedforeground region. Referring to the example illustrated in FIG. 3, theforeground extraction unit 401 in the camera adapter 113 connected tothe camera 103 extracts the foreground region from the image 250 andgenerates a foreground image 252. Referring to the example illustratedin FIG. 4, the foreground extraction unit 401 in the camera adapter 111connected to the camera 101 extracts the foreground region from theimage 253 and generates a foreground image 255.

For example, the foreground extraction unit 401 extracts the foregroundregion by acquiring the difference between a pre-generated imagerepresenting the background (hereinafter referred to as a backgroundimage) and the captured image. Examples of methods for extracting theforeground region include a method of acquiring the difference betweenimaging frames captured in succession, a method of identifying theforeground region by using machine learning, and other various methods.Examples of methods for calculating the difference include the use ofcolor difference and luminance difference. As the foreground image, theforeground extraction unit 401 generates texture data having colorinformation for an object and silhouette data not including the colorinformation for the object. The silhouette data is generated by settingthe image value of the foreground region to 1, and setting the pixelvalue of the region other than the foreground region to 0.

The foreground extraction unit 401 according to the present exemplaryembodiment generates rectangular image data including the foregroundregion as the foreground image. However, the foreground extraction unit401 can also generate a foreground image by clipping along the outlineof the foreground. More specifically, the foreground image is onlyrequired to include at least pixel values corresponding to theforeground region. Referring to the examples illustrated in FIGS. 3 and4, the object corresponding to the foreground in the captured image isonly the player 202. However, a plurality of foregrounds may be includedin the captured image. If a plurality of foregrounds is included in thecaptured image, a foreground image is generated for each foreground.Referring to the examples in FIGS. 3 and 4, the image 250 has a largerforeground region in the captured image than that in the image 253.Thus, also for the generated foreground image, the foreground image 252has a larger data amount than the foreground image 255.

The camera information acquisition unit 402 acquires information aboutcamera and lens settings (hereinafter referred to as camerainformation). The camera information acquisition unit 402 acquires thecamera information from the camera 101 directly connected to the cameraadapter 111 itself and the storage unit, such as the auxiliary storagedevice 1404 in the camera adapter 111. The camera informationacquisition unit 402 can transmit the camera information acquired fromthe camera 101 to the control server 300 and acquire the camerainformation from the control server 300. This configuration enables thecamera information acquisition unit 402 to acquire, via the controlserver 300, the camera information related to the camera 101 notdirectly connected to the camera adapter 111 itself. The camerainformation includes, for example, information about the followingitems:

-   -   (1) The sensor size of the camera 101;    -   (2) The type of the lens 121 included in the camera 101;    -   (3) The focal length of the lens 121 included in the camera 101;    -   (4) The imaging range of the lens 121 included in the camera        101;    -   (5) The exposure of the camera 101;    -   (6) The installation position, the image capturing direction,        and the distance from the imaging region of the camera 101;    -   (7) The transmission destination and the communication path of        image data based on image capturing by the camera 101;    -   (8) The priority of image data based on image capturing by the        camera 101; and    -   (9) The data amount of image data based on image capturing by        the camera 101.

The camera information acquisition unit 402 can acquire, for example,the information about the items (1) to (6) from the camera 101 directlyconnected to the camera information acquisition unit 402. The camerainformation acquisition unit 402 prestores the information about theitem (7), for example, in the storage unit, such as the auxiliarystorage device 1404, and acquires the stored information from thestorage unit. The information about the item (8) is used for determiningthe priority of transmission of the image data generated based on thecaptured image to be transmitted from the camera 101, in comparison withother one of the camera adapters 111 to 114. The priority is prestoredin the respective camera adapters 111 to 114 or determined by thecontrol server 300 and then transmitted to the camera informationacquisition unit 402 in the respective camera adapters 111 to 114. Theinformation about the item (9) is acquired, for example, by theforeground extraction unit 401 calculating the data amount of the imagedata at the time of image data generation and then transmitting the dataamount to the camera information acquisition unit 402. The camerainformation acquisition unit 402 can store and reference the data amountof the image data based on image capturing performed in the past, in theauxiliary storage device 1404. The camera information acquisition unit402 can acquire the data amount of the image data to be transmitted byeach of the camera adapters 111 to 114, from the control server 300.

The above-described information included in the camera information is tobe considered as merely an example and is not limited to this example.For example, the camera information may include only arbitraryinformation out of the information about the items (1) to (9) or includeinformation different from the information about the items (1) to (9).

The communication parameter acquisition unit 403 acquires communicationparameters determined based on the camera information acquired from thecamera information acquisition unit 402. The communication parametersare information for controlling image data communication and are usedfor processing that is performed by the communication unit 405 and thedata amount reduction unit 404 (described below). The communicationparameters according to the present exemplary embodiment are determinedby the control server 300 and then transmitted to the communicationparameter acquisition unit 403 in the respective camera adapters 111 to114 via the communication path. The communication parameter acquisitionunit 403 may be configured to acquire the communication parameters fromthe control server 300 without using the communication path. Thecommunication parameters can include various information in accordancewith the image data communication method. Examples of communicationmethods and communication parameters will be described below.

The data amount reduction unit 404 reduces the data amount of the imagedata generated by the foreground extraction unit 401, based on thecommunication parameters acquired by the communication parameteracquisition unit 403. The data amount reduction unit 404 can calculatethe communicable data amount by using the information included in thecommunication parameters. Thus, the data amount reduction unit 404performs processing for fitting the data amount of the image datagenerated by the foreground extraction unit 401 into the calculatedcommunicable data amount. Examples of processing for reducing the dataamount include the following pieces of processing:

-   -   Changing quantization parameters to be used in image data        compression or changing the compression method from lossless        compression to lossy compression to achieve higher compression        rate;    -   Deleting foreground images in ascending order of size of the        foreground regions;    -   Deleting foreground images in descending order of distance        thereof to the center of the captured image;    -   Presetting priority for each partial region in the imaging        region, and deleting foreground images corresponding to an        object included in the respective partial regions in ascending        order of priority of the partial region;    -   Presetting priority for each object (e.g., for each of a        plurality of players 202), and deleting foreground images        corresponding to the respective objects in ascending order of        priority of the object;    -   Deleting foreground images corresponding to an object in        descending order of distance of the object from a specific        object; and    -   Deleting all foreground images.

The data amount reduction unit 404 performs at least one piece ofarbitrary processing out of the above-described processing to fit thedata amount into the communicable data amount. Which processing is to beperformed by the data amount reduction unit 404 may be preset for eachof the camera adapters 111 to 114 or specified by a user operation onthe operation unit 1406. In a case where the data amount of the imagedata falls within the communicable data amount, the data amountreduction unit 404 does not reduce the data amount. However, ageneration region for generating a three-dimensional model may be presetfor the imaging region, and the foreground images corresponding toobjects outside the generation region may be deleted regardless of thedata amount. Alternatively, the foreground extraction unit 401 may beconfigured not to generate foreground images corresponding to objectsoutside the generation region. This enables the image processing server100 to communicate image data that is not used to generate a virtualviewpoint image and to prevent the communication path from beingunnecessarily used. The data amount reduction unit 404 may be configuredto store original image data before being subjected to the processing ofreducing the data amount, in the auxiliary storage device 1404. Thisenables the camera adapter 111 to transmit high-quality image databefore being subjected to the data amount reduction, at the timing whenlimitations on the communication bandwidth are alleviated afterwards.

The communication unit 405 performs image data communication inconformance with a predetermined communication method based on thecommunication parameters acquired by the communication parameteracquisition unit 403. The communication unit 405 also communicate suchimage data as the background image in addition to the foreground imagegenerated by the foreground extraction unit 401.

A function configuration of the control server 300 will be describedbelow with reference to FIG. 5. The control server 300 includes aninformation acquisition unit 500, a communication parameterdetermination unit 501, and a control unit 502. Processing units of thecontrol server 300 will be described below.

The information acquisition unit 500 acquires the camera informationtransmitted from the plurality of camera adapters 111 to 114. Thecommunication parameter determination unit 501 determines thecommunication parameters based on the camera information acquired by theinformation acquisition unit 500. The control unit 502 controls theplurality of camera adapters 111 to 114 to perform communication basedon the communication parameters determined by the communicationparameter determination unit 501. In controlling the camera adapters 111to 114, the control unit 502 also transmits the camera information andthe communication parameters acquired from the camera adapters 111 to114. This enables the respective camera adapters 111 to 114 to grasp thecamera information about the corresponding one of the cameras 101 to 104not directly connected thereto, and perform processing of reducing thedata amount based on the communication parameters. The control unit 502issues instructions to start and end communication performed by therespective camera adapters 111 to 114 and performs processing ofchanging the transmission destination and the communication path of theimage data that the camera adapters 111 to 114 communicate.

The control unit 502 controls not only communication performed by therespective camera adapters 111 to 114 but also the cameras 101 to 104,the lenses 121 to 124, and the image processing server 100. Asprocessing for controlling the cameras 101 to 104, the control unit 502performs processing of starting and ending image capturing, changing thecamera parameters, such as the exposure and shutter speed, andmonitoring changes of the camera parameters. As processing ofcontrolling the lenses 121 to 124, the control unit 502 performsprocessing of adjusting the focal length and focus and monitoringchanges of the focal length and focus. As processing of controlling theimage processing server 100, the control unit 502 controls the start andend of processing of generating a virtual viewpoint image.

Control processing performed by the control unit 502 is not limited tothe above-described processing. The control unit 502 may perform othercontrol processing, monitoring of each apparatus, and informationacquisition. The control unit 502 may control the camera 101 and thelens 121 via the camera adapter 111 or directly control the imagingapparatus and the lens 121. At least part of processing of controllingthe camera adapter 111, the camera 101, the lens 121, and the imageprocessing server 100 performed by the control unit 502 may be performedby the camera adapter 111, the camera 101, and the image processingserver 100, respectively.

The function configurations of the camera adapters 111 to 114 and thecontrol server 300 have been described above. The communication methodand communication parameters to be used by the image processing system10 according to the present exemplary embodiment will be described indetail below. A description will be provided of a communication methodthat is performed by the control server 300 to control communication ofa plurality of image data pieces by using the plurality of cameraadapters 111 to 114 based on image capturing conditions of the pluralityof cameras 101 to 104.

<Time-Division Multiplex Method>

An example where the camera adapters 111 to 114 communicates image databy using the time-division multiplex method in the image processingsystem 10 will be described below. FIGS. 6A and 6B are timing chartsillustrating the timing of image data transmission in a case where thecamera adapters 111 to 114 perform communication by using thetime-division multiplex method. The timing of image data transmission inthe initial state will be described below with reference to FIG. 6A.

Referring to FIG. 6A, the times T1-0, T2-0, . . . indicate the timingwhen the camera 101 performs image capturing. The times T1-0, T2-0, . .. each correspond to the frame rate of image capturing by the camera101. For example, when the camera 101 performs image capturing at 60frames per second (fps), the time period between the times T1-0 and T2-0is 1/60 second. The times T1-1, T1-2, T1-3, . . . are the timedetermined by dividing the time period between the times T1-0 and T2-0by the number of cameras. The times T2-1, T2-2, T2-3, . . . are alsodetermined in a similar way. According to the present exemplaryembodiment, since four cameras 101 to 104 are provided, the time periodfrom the time T1-0 to the time T1-1 is 1/240 second at a frame rate of60 fps.

In each of the camera adapters 111 to 114, the timing of starting imagedata transmission and the time length that can be used for image datacommunication are preset as initial values. For example, assuming thatthe time T1-0 is 0 seconds, the time T1-1 corresponds to 1/240 second,the time T1-2 corresponds to 1/120 second, the time T1-3 corresponds to1/80 second, and the time T2-0 corresponds to 1/60 second. The time T2-1and subsequent times are determined in a similar way. Each of the cameraadapters 111 to 114 starts image data transmission to the downstreamcamera adapter at the time specified by the control server 300. In thiscase, the time length during which the communication path for image datacommunication can be used refers to the time period since the cameraadapter starts the transmission of the image data based on imagecapturing by the camera directly connected to the camera adapter itselfuntil the adjacent downstream camera adapter starts image datatransmission similarly. Although not illustrated in FIGS. 6A and 6B,each of the camera adapters 111 to 114 transmits image data transmittedfrom the corresponding upstream one of the camera adapters 111 to 114 inthe time period during which it is not transmitting image data based onimage capturing by the corresponding one of the cameras 101 to 104directly connected thereto. For example, the camera adapter 112 acquiresimage data 1001 a transmitted from the camera adapter 111 and transmitsthe acquired image data 1001 a to the camera adapter 113 during the timeperiod between the times T1-0 and T1-1. At this time, a delay can occurin the time period between image data acquisition and image datatransmission. The transmission of the image data based on imagecapturing by the respective cameras 101 to 104 connected to thecorresponding one of the camera adapters 111 to 114 itself is delayedfrom the start time of transmission. However, since these delays areextremely short, the following descriptions will be made on the premisethat no delay occurs.

Although, in the present exemplary embodiment, the timing of image datatransmission is determined by dividing the time period from the timewhen the plurality of cameras 101 to 104 performs image capturing to thetime when the following image capturing is performed. The configurationis not limited thereto. For example, the start time of image datatransmission by the camera adapter 111 does not necessarily need tocoincide with the times T1-0, T2-0, . . . The time length that can beused for image data communication (a time period of 1/60 second in thiscase) may be acquired by dividing the time period identified based onthe frame rate of image capturing performed by the plurality of cameras101 to 104.

Referring to FIG. 6A, the camera adapter 111 starts transmission of theimage data 1001 a based on image capturing by the camera 101 connectedto the camera adapter 111 itself at the time T1-0, and completes imagedata transmission to the downstream camera adapter 112 during the timeperiod before the time T1-1. Similarly, the camera adapter 112 startsthe transmission of image data 1002 a based on image capturing by thecamera 102 connected to the camera adapter 112 itself at the time T1-1,and completes image data transmission to the downstream camera adapter113 during the time period before the time T1-2. This also applies tothe camera adapter 113. The camera adapter 114 at the end of the daisychain connection starts the transmission of image data 1004 a at thetime T1-3, and completes image data transmission to the image processingserver 100 during the time period before the time T2-0 at which thefollowing image capturing is performed. By performing communication inthis way, the plurality of camera adapters 111 to 114 can complete imagedata transmission corresponding to all of the cameras 101 to 104 duringthe time period from when image capturing is started to when thefollowing image capturing is performed by the respective cameras 101 to104. After the time T2-0, the image data communication is also performedin a similar way.

At this time, the data amount reduction unit 404 calculates the dataamount communicable in 1/240 seconds for the image data to betransmitted by each of the camera adapters 111 to 114, based on thetransmission bandwidth of the communication path. If the data amount ofthe image data exceeds the calculated communicable data amount, the dataamount reduction unit 404 performs processing of reducing the dataamount. The image data transmission may be performed without using allof 1/240 seconds. For example, if the data amount of the image data issmaller than the calculated communicable data amount, image datatransmission may be completed in a time period shorter than 1/240seconds. Also, in this case, each of the camera adapters 111 to 114 doesnot change the start time of image data transmission.

FIG. 6B illustrates an example in which the time length to be used forimage data transmission is changed based on the focal length of therespective lenses 121 to 124. Referring to the example illustrated inFIG. 6B, the focal length of the lens 123 is changed to be longer thanthe focal lengths of other lenses 121, 122, and 124. As described above,the data amount of the foreground image data may increase withincreasing focal length of the respective lenses 121 to 124. Thus, ifimage data communication is performed based on the communication timingin the initial state, image data based on image capturing by the camera103 may be lost in a time period of 1/240 seconds. The image data basedon image capturing by other cameras 101, 102, and 104 may become smallerin data amount than the image data for the camera 103 and therefore mayhave been transmitted in a time period shorter than 1/240 second.However, the time when each of the camera adapters 111 to 114 startsimage data transmission remains unchanged. In such a situation, thecommunication path cannot be effectively utilized, possibly resulting insuch an issue, the degradation of the communication efficiency and theoccurrence of missing image data. Thus, the control server 300 changesthe time length that can be used for image data transmission by each ofthe camera adapters 111 to 114, based on the focal length of therespective lenses 121 to 124.

The information acquisition unit 500 in the control server 300 acquires,as imaging conditions, information about the focal length of each of thecameras 101 to 104 (the information about item (3) included in thecamera information) and transmits the information to the communicationparameter determination unit 501. The communication parameterdetermination unit 501 determines the time when each of the cameraadapters 111 to 114 starts image data transmission and the time lengththat can be used for image data communication by each of the cameraadapters 111 to 114, as communication parameters. For example, in a casewhere the focal length of the lens 123 is twice the focal lengths ofother lenses, the communication parameter determination unit 501 changesthe time length that can be used for image data transmission by thecamera adapter 113 to be twice the time lengths for other cameraadapters. The communication parameter acquisition unit 403 divides thetime period between the times T1-0 and T2-0 based on the determined timelength to determine the time period during which each camera adapter 111performs image data transmission. Thus, as illustrated in FIG. 6B, thetime length that can be used for image data transmission by the cameraadapter 113 is twice the time lengths for other camera adapters. Theratio of the time length that can be used for image data transmissiondoes not necessarily need to be identical to the ratio of the focallength. The time length is determined so that at least the time lengthof the transmission of the image data based on image capturing by one ofthe cameras 101 to 104 that has a longer focal length becomes longerthan the time lengths for the other one of the cameras 101 to 104.

Thus, the control server 300 determines the time length during which thecommunication path can be used for image data communication, based onthe focal length of the respective lenses 121 to 124 included in thecorresponding one of the cameras 101 to 104. The communicationparameters can be determined based on various information included inthe camera information, in addition to the focal length. For example,the communication parameter determination unit 501 estimates the dataamount of the generated foreground image as an imaging condition basedon the information about items (1) and (6) included in the camerainformation. It is estimated that the data amount of the image dataincreases with increasing sensor size. It is also estimated that theforeground region enlarges and the data amount of the image dataincreases with decreasing distance between the respective cameras 101 to104 and the imaging region.

The communication parameter determination unit 501 can determine thetime length during which the communication path can be used for imagedata communication, based on the ratio of the data amount estimated foreach of the cameras 101 to 104. This enables ensuring a longer timelength for the camera adapters that can transmit a larger amount ofimage data.

The distance between the camera 101 and the imaging region can beidentified based on the information about the installation position, theimage capturing direction, and the focal length of the camera 101. Thus,the control server 300 acquires information that enables identificationof the distance between the camera 101 and the imaging region todetermine the time length.

The communication parameter determination unit 501 can also acquire theinformation about the item (9) included in the camera information and,based on the data amount of the past image data, estimate the dataamount of the image data to be generated in the following imagecapturing. In a case where image capturing is performed in succession asin moving image capturing, the position of an object hardly quicklychanges in a short time between image capturings. Thus, since it isestimated that the data amount of the foreground image also changessmall, the communication parameter determination unit 501 can estimatethe data amount of the image data to be generated in the following imagecapturing.

The communication parameter determination unit 501 may determine thecommunication parameters as imaging conditions based on the informationabout the item (4) included in the camera information. For example, in acase where the imaging range to be captured by the camera 101 includes aregion outside the generation region for generating three-dimensionalgeometric data, the foreground image corresponding to the objectexisting in the region outside the generation region can be deleted bythe data amount reduction unit 404 or generation thereof can be omitted.In this configuration, the communication parameters are determinedtaking into account the data amount of the foreground image to bedeleted or not to be generated.

Examples where the data amount is estimated based on the imaging rangeto be captured by the respective cameras 101 to 104 will be describedbelow with reference to FIGS. 9 and 10. FIG. 9 illustrates examples ofarrangements of the cameras 101 and 102. FIG. 10 illustrates examples ofcaptured images acquired by the cameras 101 and 102 in the examplesillustrated in FIG. 9. Referring to FIG. 10, captured images 132 and 131are acquired by the cameras 102 and 101, respectively.

Referring to FIG. 9, a field 150 indicates the generation region forwhich three-dimensional geometric data is to be generated by the imageprocessing server 100. More specifically, three-dimensional geometricdata is generated for the object present in the field 150, and notgenerated for the object present outside the field 150. Referring toFIG. 9, the imaging range 132 of the camera 102 is included in the field150. The imaging range 131 captured by the camera 101 includes a range140 included in the field 150 and a range 141 not included in the field150. Referring to FIG. 9, a player 202 is present in the field 150, anda player 203 is present outside the field 150. More specifically, evenafter the camera adapter 111 generates a foreground image 259corresponding to the player 203 and transmits the image to the imageprocessing server 100, three-dimensional geometric data is notgenerated. In such a case, the camera adapter 111 reduces the dataamount of the image data to be communicated, by deleting or notgenerating the foreground image corresponding to the player 203.

Thus, in a case where there is an imaging apparatus that captures animage of a region not subjected to generation of three-dimensionalgeometric data, the communication parameter determination unit 501determines the communication parameters based on the size of the rangeincluded in the region for which three-dimensional geometric data is notgenerated. For example, referring to the example in FIG. 9, if the rangeincluded in the region for which three-dimensional geometric data isgenerated is 50% of the entire imaging range of the camera 101. In thiscase, the communication parameter determination unit 501 sets the timelength that can be used for image data transmission by the camera 101 to50% of the time length that can be used by the camera 102. Thus, thecontrol server 300 can determine the communication parameters based onthe imaging range to be captured by the respective cameras 101 to 104.

The imaging range to be captured by the respective cameras 101 to 104may also be identified based on the information other than theinformation about the item (4) included in the camera information. Forexample, in a case where there is not the information about the item(4), the imaging range can be identified based on the information aboutthe installation position, the image capturing direction, and the focallength of the respective camera 101 to 104. Thus, the control server 300acquires information that enables identification of the imaging range todetermine the time length.

The camera adapter 111 performs communication under the control of thecontrol server 300. At this time, the data amount reduction unit 404 inthe camera adapter 111 calculates the data amount that can betransmitted in the time length determined by the control server 300 andreduces the data amount of the image data to fit the image data into thecalculated data amount.

In time-division multiplex communication, the control server 300determines the time lengths during which the communication path can beused for image data communication, based on the camera information, andcontrols communication by the camera adapters 111 to 114. This enablesimage data communication with effective utilization of the communicationpath. The control server 300 first determines the communicationparameters based on the camera information acquired when the imageprocessing system 10 is activated. In the following processing, thecontrol server 300 periodically acquires the camera information anddynamically changes the communication parameters when the camerainformation is changed. This configuration also applies to the followingcommunication methods. The configuration is not limited thereto. Thecommunication parameters determined first may be used by when the imageprocessing system 10 is deactivated. In this case, when imagingconditions, such as the focal length of the respective camera 101 to104, remain unchanged from the initial values, the communicationparameters determined based on the initial values are preset to each ofthe camera adapters 111 to 114. This configuration enables reduction inthe processing load relating to the determination of the communicationparameters. This configuration is also applicable to the bandwidthmultiplex method and the token passing method (described below).

<Bandwidth Multiplex Method>

An example where the camera adapters 111 to 114 perform image datacommunication by using the bandwidth multiplex method in the imageprocessing system 10 will be described below. FIGS. 7A and 7B are timingcharts each illustrating the timing when the camera adapters 111 to 114perform image data communication by using the bandwidth multiplexmethod. The timing of image data transmission in the initial state willbe described below with reference to FIG. 7A.

The times T1-0, T2-0, . . . are similar to the times in FIGS. 6A and 6B.Each of the camera adapters 111 to 114 performs image data transmissionby using the bandwidth determined by dividing the communicationbandwidth of the communication path by the number of cameras. Forexample, in a case where the communication bandwidth of thecommunication path is 10 G bits per second (bps), the camera adapters111 to 114 perform image data transmission at a communication throughput(transmission rate) of 2.5 G bps, which is obtained by dividing 10 Ginto quarters. More specifically, the camera adapters 111, 112, 113, and114 divide image data 2001 a, 2001 b, 2001 c, and 2001 d, respectively,generated by themselves into communication packets, and perform packettransmission at the frequency determined based on the determinedthroughput. The communication throughput may be determined based notonly on the communication bandwidth of the communication path but alsoon the image data reception performance of the camera adapters 111 to114.

FIG. 7B illustrates an example in which the image data throughput ischanged based on the focal length of the respective lenses 121 to 124.Referring to the example illustrated in FIG. 7B, the focal length of thelens 123 is changed to be longer than the focal lengths of the otherlenses 121, 122, and 124. When the focal lengths of the lenses 121 to124 differ for a reason similar to that for the time-division multiplexmethod, the communication path may not be effectively utilized at thetime of image data transmission. Thus, the control server 300 determinesthe time length that can be used for image data communication by each ofthe camera adapters 111 to 114, based on the focal length of therespective lenses 121 to 124. The control server 300 determines thefrequency of packet transmission performed by each of the camera adapter111 to 114, based on the determined time length, and controls the cameraadapters 111 to 114 to perform packet communication at the determinedfrequency. The frequency of packet transmission is represented by thenumber of packets to be transmitted in unit time.

The information acquisition unit 500 in the control server 300 acquiresthe information about the focal length of each of the cameras 101 to 104(the information about the item (3) included in the camera information)and transmits the information to the communication parameterdetermination unit 501. The communication parameter determination unit501 determines the throughput of image data transmission by each of thecamera adapters 111 to 114 as a communication parameter. For example,when the focal length of the lens 123 is twice the focal lengths ofother lenses, the communication parameter determination unit 501 changesthe throughput for the camera adapter 113 to transmit image data to betwice the throughputs of other camera adapters. As illustrated in FIG.7B, the throughput that can be used for image data transmissionperformed by the camera adapter 113 becomes twice the throughputs ofother camera adapters. The ratio of the throughput that can be used forimage data transmission does not necessarily need to be identical to theratio of the focal length. The throughput is determined so that at leastthe throughput corresponding to one, out of the cameras 101 to 104,having a longer focal length becomes larger than the throughputs ofother cameras. The control server 300 controls the frequency of imagedata packet transmission performed by each of the camera adapters 111 to114 based on the determined throughput.

As in the time-division multiplex method, the control server 300 candetermine the throughput based on various information (e.g., theinformation about the times (1), (4), (6) and (9)) included in thecamera information. The camera adapters 111 to 114 perform communicationunder the control of the control server 300. At this time, the dataamount reduction unit 404 in the respective camera adapters 111 to 114calculates the data amount that can be transmitted during the timeperiod from when image capturing is performed to when the followingimage capturing is performed based on the throughput determined by thecontrol server 300. The data amount reduction unit 404 reduces the dataamount of the image data to fit the image data into the calculated dataamount.

Thus, in bandwidth multiplex communication, the control server 300determines the throughput that can be used for image data communication,based on the camera information, and controls communication by thecamera adapters 111 to 114. This enables image data communication thateffectively utilizes the communication path.

<Token Passing Method>

A description will be provided of an example in which the cameraadapters 111 to 114 perform image data communication by using the tokenpassing method in the image processing system 10. FIGS. 8A and 8B aretiming charts each illustrating the timing of image data transmission ina case where the camera adapters 111 to 114 perform communication byusing the token passing method. The timing of image data transmission inthe initial state will be described below with reference to FIG. 8A.

The times T1-0, T2-0, . . . are similar to the times in FIGS. 6A, 6B,7A, and 7B. Referring to the example in FIG. 8A, image data transmissionis performed by the camera adapters 111, 112, 113, and 114 in thisorder. Each of the camera adapters 111, 112, 113, and 114 starts imagedata transmission after completion of previous image data transmission.The example in FIG. 8A illustrates that the camera adapter 114 does notcomplete the transmission of image data 3004 a by the time T2-0 at whichthe following image capturing is to be performed. In this case, thecamera adapter 114 stops transmission of the image data 3004 a or doesnot transmit the entire image data 3004 a. Thus, part or whole of theimage data 3004 a will be lost. In this way, even important image datamay possibly be lost depending on the order of image data transmission.

FIG. 8B illustrates an example in which the order of image datatransmission is changed based on the camera information. Although thecamera information can include the preset priority for each of thecameras 101 to 104 (the information about the item (8) included in thecamera information), the control server 300 can determine the priorityas a communication parameter in consideration of other informationincluded in the camera information.

FIG. 8B illustrates an example where the order of image datatransmission is changed based on the camera information. Referring tothe example in FIG. 8B, since the lens 121 has a long focal length, thedata amount of the image data 3001 b to be transmitted by the cameraadapter 111 is larger than the data amounts for other camera adapters112, 113, and 114. Assume that the camera 102 captures a partial regionhaving lower priority than partial regions for other cameras 101, 103,and 104. For example, the communication parameter determination unit 501determines the priority based on the information about the focal lengthincluded in the camera information acquired by the informationacquisition unit 500. The communication parameter determination unit 501prestores a table in which the focal length is in association with thepriority in the auxiliary storage device 1404, and refers to the tableto determine the priority. The control server 300 determines thetransmission timing for each piece of image data based on the determinedpriority. Thus, the image data 3001 b is transmitted by the cameraadapter 111 in a prioritized way. The order of the transmission of theimage data based on image capturing by the ones having the same focallength out of the cameras 101 to 104 is determined based on the presetpriority (the information about the item (8) included in the camerainformation). As described above, important image data is transmitted ina prioritized way. This prevents important image data from being lost.

The communication parameter determination unit 501 may include a tablein which the information about the items (1), (4), (6), and (9) includedin the camera information is in association with the priority inaddition to the focal length. The communication parameter determinationunit 501 may store a table in which each piece of the camera informationis in association with a value indicating the priority, and determinethe priority so that corresponding image data transmission is performedin descending order of the sum of the values indicating the priority.Referring to the example in FIG. 8B, although the priority determined atthe time T1-0 is also applied to the time T2-0 and subsequent time, thisis not restrictive. The priority may be dynamically changed according tochanges of the camera information.

Here, a camera adapter, out of the camera adapters 111 to 114, that isunable to complete image data transmission by the time of the followingimage capturing may take measures by stopping or not performing imagedata transmission. However, this is not restrictive. For example, thedata amount reduction unit 404 determines that image data transmissioncannot be completed based on the communication parameters and calculatesthe data amount that can be transmitted during the time period by thetime when the following image capturing is performed. The data amountreduction unit 404 reduces the data amount of the image data so as tofit the image data into the data amount resulting from the calculation.

In this way, in the communication using the token passing method, thecontrol server 300 determines the priority of image data transmissionbased on the camera information, and controls communication by thecamera adapters 111 to 114. This enables important image data to beprevented from being lost. The processing of determining the priorityand controlling the order of image data transmission is also applicableto the time-division multiplex method.

The foregoing are descriptions of methods of determining thecommunication parameters in a case where each of the time-divisionmultiplex, the bandwidth multiplex, and the token passing methods isused as the communication method in the image processing system 10. Inthe image processing system 10 according to the present exemplaryembodiment, the communication method to be used is preset based on auser specification or the like. From the time of activation, the imageprocessing system 10 performs communication based on the presetcommunication method.

FIG. 11 is a flowchart illustrating processing performed by the cameraadapter 111 and the control server 300 according to the presentexemplary embodiment. The processing in FIG. 11 is implemented by eachof the CPUs of the respective camera adapters 111 to 114 and the controlserver 300 reading a program stored in the ROM or the auxiliary storagedevice and then executes the program. When the image processing system10 is activated based on a user instruction, processing is started.

In step S1121, the camera information acquisition unit 402 in therespective camera adapters 111 to 114 acquire the camera informationfrom the corresponding one of the cameras 101 to 104 and transmit theinformation to the control server 300. In step S1111, the control server300 initializes the entire image processing system 10. Here, the controlserver 300 determines the communication method to be used by the imageprocessing system 10. The communication parameter determination unit 501in the control server 300 determines the initial values of thecommunication parameters as initial settings, based on the camerainformation transmitted from the respective camera adapters 111 to 114.If the respective camera adapters 111 to 114 do not perform theoperation in step S1121 and hence the control server 300 does notacquire the camera information, the control server 300 makes setting toenable communication in the initial state by the communication method.

In step S1112, the control server 300 transmits an instruction to startimage capturing to the respective camera adapters 111 to 114 based on auser input. When the instruction to start image capturing is transmittedfrom the control server 300, then in step S1122, the respective cameraadapters 111 to 114 control the corresponding one of the cameras 101 to104 directly connected thereto to start image capturing. In step S1123,the camera information acquisition unit 402 in the respective cameraadapters 111 to 114 periodically acquires the camera information fromthe corresponding one of the cameras 101 to 104 while the respectivecameras 101 to 104 are performing image capturing. The frequency atwhich the camera information acquisition unit 402 acquires the camerainformation is determined, for example, at the time of initial settingin step S1111. The camera information acquisition unit 402 transmits theacquired camera information to the control server 300.

In step S1113, the information acquisition unit 500 in the controlserver 300 acquires the camera information transmitted from therespective camera adapters 111 to 114, and transmits the information tothe communication parameter determination unit 501. The communicationparameter determination unit 501 determines whether the acquired camerainformation has been changed from the currently stored camerainformation. If the acquired camera information has been changed (YES instep S1113), the processing proceeds to step S1114. In step S1114, thecommunication parameter determination unit 501 determines thecommunication parameters based on the information about the items (1),(3), (4), (6), (8), and (9) included in the camera information. Thecommunication parameters determined here include the time length in thetime-division multiplex method, the throughput in the bandwidthmultiplex method, and the priority in the token passing method. Thecontrol unit 502 transmits the determined communication parameters tothe respective camera adapters 111 to 114, and controls the respectivecamera adapters 111 to 114 to perform communication based on thecommunication parameters. On the other hand, if the communicationparameter determination unit 501 determines that the acquired camerainformation has not been changed (NO in step S1113), the operation instep S1114 is skipped.

In step S1124, the image acquisition unit 400 in the respective cameraadapters 111 to 114 acquire the captured image from the correspondingone of the cameras 101 to 104. The foreground extraction unit 401extracts the foreground region from the captured image acquired by theimage acquisition unit 400 and generates image data (foreground image)including texture data and silhouette data for the object. In stepS1125, the data amount reduction unit 404 determines whether the dataamount of the generated image data exceeds the communicable data amountdetermined based on the communication parameters. At this time, if thecommunication parameters have been transmitted from the control server300 in step S1114, the data amount reduction unit 404 uses the acquiredcommunication parameters for this determination. If the data amountreduction unit 404 determines that the data amount of the generatedimage data exceeds the communicable data amount (YES in step S1125), theprocessing proceeds to step S1126. In step S1126, the data amountreduction unit 404 performs a process of reducing the data amount of theimage data to fit the image data into the communicable data amount. Ifthe data amount reduction unit 404 determines that the data amount ofthe generated image data does not exceed the communicable data amount(NO in step S1125), the operation in step S1126 is skipped.

In step S1127, the communication unit 405 in the respective cameraadapters 111 to 114 transmits the image data to the downstream cameraadapter or the image processing server 100 under the control of thecontrol server 300. In step 51115, the control server 300 determineswhether an instruction to end image capturing is issued by a user input.If the control server 300 determines that the instruction is issued (YESin step S1115), the processing proceeds to step S1116. In step S1116,the control server 300 transmits the instruction to end image capturingto the respective camera adapters 111 to 114. If the control server 300determines that the instruction is not issued (NO in step S1115), theprocessing returns to step S1113. The control server 300 performs theoperations in step S1113 and subsequent steps again. In step S1128, therespective camera adapters 111 to 114 determine whether an instructionto end image capturing is issued from the control server 300. If therespective camera adapters 111 to 114 determine that the instruction isissued (YES in step S1128), the processing proceeds to step S1129. Instep S1129, the respective camera adapters 111 to 114 control thecorresponding one of the cameras 101 to 104 to end image capturing. Ifthe control server 300 determines that the instruction to end imagecapturing is not issued (NO in step S1128), the processing returns tostep S1123. The respective camera adapter 111 to 114 perform theoperations in step S1123 and subsequent steps again. In step S1129, inresponse to the respective cameras 101 to 104 ending image capturing,the control server 300 and the respective camera adapters 111 to 114 endprocessing.

The control server 300 according to the present exemplary embodimentdetermines the time length that can be used for communication of aplurality of pieces of image data by the respective camera adapters 111to 114, based on the imaging conditions of the corresponding one of thecameras 101 to 104, and performs control so that communication using thepredetermined communication path is performed in the determined timelength. This enables the control server 300 to effectively utilize thecommunication path and prevent the degradation of the communicationefficiency and the occurrence of a loss of image data.

A second exemplary embodiment of the present disclosure will bedescribed below. In the first exemplary embodiment, a description hasbeen provided of an example in which all of the cameras 101 to 104perform image capturing at the same frame rate. In the present exemplaryembodiment, a description will be provided of a communication method ina case where a plurality of cameras 101 to 104 includes one(s) havingdifferent frame rates. The configuration of the image processing system10, and the function and hardware configuration of each apparatusincluded in the image processing system 10 are similar to those in thefirst exemplary embodiment, and redundant descriptions thereof will beomitted.

FIGS. 12A and 12B illustrate methods in which a plurality of cameras 101to 104 including one having different imaging frame rates transmitsimage data based on the time-division multiplex method. Referring to theexample in FIGS. 12A and 12B, the cameras 101 and 103 perform imagecapturing at 60 fps, and the cameras 102 and 104 perform image capturingat 30 fps. The time period between the times T1-0 and T2-0 is 1/60second. The cameras 101 and 103 perform image capturing at the timesT1-0, T2-0, T3-0, . . . On the other hand, the cameras 102 and 104perform image capturing at the times T1-0, T3-0, . . . , since theimaging frame rate is 30 fps.

FIG. 12A illustrates an example where each of the camera adapters 111 to114 performs communication in synchronization with image capturingperformed at 60 fps. Referring to the example in FIG. 12A, the cameraadapters 111 to 114 perform image data communication during the timeperiod between the time T1-0 at which image capturing is performed andthe time T2-0 at which the following image capturing is performed. Thetime length that can be used for communication is also determined sothat four pieces of image data are to be transmitted. On the other hand,between the times T2-0 and T3-0, the camera adapters 112 and 114 do nottransmit image data because the cameras 102 and 104 do not perform imagecapturing. Thus, the time length is determined so that two pieces ofimage data are transmitted between the times T2-0 and T3-0. Ifcommunication is performed in this way, the number of pieces of imagedata to be transmitted is larger than that at other time, in a casewhere the timing of image capturing by the cameras 101 to 104 issynchronized, and hence a loss of image data is likely to occur. Thecontrol server 300 according to the present exemplary embodimentdetermines the timing at which each of the camera adapters 111 to 114transmits image data, thus preventing the occurrence of this issue.

Referring to the example illustrated in FIG. 12B, as in FIG. 12A, thecameras 101 and 103 perform image capturing at the times T1-0, T2-0,T3-0, . . . The cameras 102 and 104 perform image capturing at the timesT1-0, T3-0, . . . Referring to the example illustrated in FIG. 12B,however, the timing at which the camera adapter 114 transmits image datais changed from that in FIG. 12A. The camera adapters 111 to 114 performimage capturing at the time T1-0, then generate image data, and startimage data transmission at a predetermined timing. At this time, thecontrol server 300 shifts the transmission timing for either one of thepieces of image data corresponding to the cameras 102 and 104 thatperform image capturing at 30 fps. Referring to the example illustratedin FIG. 12B, the timing of image data transmission by the camera adapter114 is delayed from the timing of image data transmission by the cameraadapter 113 by 0.5 frames. By controlling the timing to start image datatransmission in this way, the control server 300 can prevent the dataamount of the image data from exceeding the data amount communicableduring a predetermined period.

As in the first exemplary embodiment, the control server 300 accordingto the present exemplary embodiment estimates the data amount for eachpiece of image data based on the camera information (e.g., the focallength). The control server 300 determines the timing of image datatransmission based on the estimated value of the data amount. At thistime, the control server 300 determines the transmission timing so thatthe data amount of the image data to be transmitted between the timesT1-0 and T2-0 becomes as equal to the data amount of the image data tobe transmitted between the times T2-0 and T3-0 as possible. In theexample in FIG. 12B, there are two cameras having an imaging frame rateof 30 fps. However, even if there are three or more cameras having thisimaging frame rate, the timing of image data transmission can bedetermined by using the above-described method. In a case where thetiming of image data transmission is shifted, the order of image datatransmission is to be determined based on the priority determined bysimilar processing to the processing according to the first exemplaryembodiment.

As discussed above, the control server 300 according to the presentexemplary embodiment determines the timing of image data transmissionbased on captured images acquired by the plurality of cameras includingcameras having different imaging frame rates. A camera having a framerate of 30 fps is used to acquire a background image such as seats andthe ground of a stadium. Since the object corresponding to theforeground is assumed to largely change in motion, it is desirable toperform image capturing to generate a foreground image by using a camerahaving a higher frame rate. However, since the background region isassumed to change smaller in motion than the object corresponding to theforeground, the image of the background region is captured by using acamera having a lower frame rate than the camera used to capture theforeground, to generate a background image. This reduces the number ofbackground images to be generated, making it possible to reduce theprocessing load. Naturally, a foreground image may be generated by usinga captured image acquired by a camera having a frame rate of 30 fps.

In the present exemplary embodiment, a description has been provided ofan example where the time-division multiplex method is used. The controlserver 300 can also determine the transmission timing similarly, even ina case where the token passing method is used. In a case where thebandwidth multiplex method is used, the control server 300 determinesthe ratio for dividing the bandwidth, based on the ratio between theimaging frame rates of the cameras 101 to 104, in addition to the methodaccording to the first exemplary embodiment.

(Other Exemplary Embodiments)

In the first and the second exemplary embodiments, a description hasbeen provided of an example in which a plurality of camera adapters in adaisy chain connection performs image data communication. However, themethod according to each exemplary embodiment is also applicable to acase where a plurality of camera adapters 111 to 114 performscommunication by using other network topologies.

Assume an example in which a plurality of camera adapters 111 to 114 areconnected to the image processing server 100 in a star connection. Insuch a case, each of the camera adapters 111 to 114 can perform directimage data transmission by using the communication path connected to theimage processing server 100. However, if a plurality of pieces of imagedata is transmitted without taking into account the data amount of theimage data or the transmission timing, the image processing server 100becomes unable to receive at least part of the plurality of pieces ofimage data transmitted from each of the camera adapters 111 to 114depending on the receiving capability of the image processing server 100and the bandwidth of the communication path. Accordingly, part or wholeof image data may be lost. Thus, also in a star connection, the loss ofimage data can be prevented by using the time-division multiplex, thebandwidth multiplex, and the token passing methods and adjusting thedata amount of the image data to be received not to exceed the receivingcapability of the image processing server 100 and the bandwidth of thecommunication path. In addition, by determining the time length that canbe used for image data communication based on the image capturingconditions (including the focal length, the imaging range, and thedistance from the imaging region) of a plurality of imaging apparatuses,the communication path can be used more efficiently with receivingcapability on the image data reception side taken into account. This isnot limited to the star connection but also applied to ‘bus’ (where eachimaging apparatus may be connected to a communication bus) and ‘mesh’(where connections may exist between any two imaging apparatuses and/orthe control server) connections.

As discussed above, the methods according to the first and the secondexemplary embodiments exhibit their advantageous effects in variousconnection methods. The above-described exemplary embodiments can alsobe applied to communication based on a combination of a plurality ofcommunication methods. An example of a combination of the time-divisionmultiplex and the bandwidth multiplex methods will be described below. Apredetermined time length in the time-division multiplex method isassigned to each of the camera adapters 111 to 114. At this time, thetime lengths each to be assigned to the respective camera adapters 111to 114 are at least partly duplicated. At the time of communication,each of the camera adapters 111 to 114 divides data into packets andtransmits packets at a predetermined frequency. In this case, at leastone of the predetermined time length and the predetermined frequency tobe assigned to the respective the camera adapters 111 to 114 aredetermined by the method according to the above-described exemplaryembodiments. Combining the time-division multiplex and the bandwidthmultiplex methods in this way enables performing communication whilepermitting the duplication of data communication timing. This makes itpossible to more flexibly handle variations in the data amount oftransmission data.

The present disclosure makes it possible to prevent a loss of image datadue to increase in the data amount of image data depending on imagingconditions.

Other Embodiments

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present disclosure has been described with reference toexemplary embodiments, it is to be understood that the disclosure is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2020-103906, filed Jun. 16, 2020, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A control apparatus comprising: an acquisitionunit configured to acquire information about an imaging condition of aplurality of imaging apparatuses; a determination unit configured todetermine a time length during which each of a plurality of pieces ofimage data based on image capturing by the plurality of imagingapparatuses is communicable, based on the information acquired by theacquisition unit; and a control unit configured to perform control sothat communication of the plurality of pieces of image data is performedin accordance with the time length determined by the determination unit.2. The control apparatus according to claim 1, wherein the imagingcondition information includes information indicating a focal length ofeach of the plurality of imaging apparatuses.
 3. The control apparatusaccording to claim 2, wherein, in a case where a focal length of a firstimaging apparatus included in the plurality of imaging apparatuses islonger than a focal length of a second imaging apparatus included in theplurality of imaging apparatuses, the determination unit determines thetime length so that a time length during which image data based on imagecapturing by the first imaging apparatus is communicable is longer thana time length during which image data based on image capturing by thesecond imaging apparatus is communicable.
 4. The control apparatusaccording to claim 1, wherein the imaging condition information includesinformation that enables identification of an imaging range to besubjected to image capturing by each of the plurality of imagingapparatuses.
 5. The control apparatus according to claim 4, wherein, ina case where a ratio of an imaging range included in a predeterminedregion out of an entire imaging range to be subjected to image capturingby a first imaging apparatus included in the plurality of imagingapparatuses is larger than a ratio of an imaging range included in thepredetermined region out of an entire imaging range to be subjected toimage capturing by a second imaging apparatus included in the pluralityof imaging apparatuses, the determination unit determines the timelength so that a time length during which image data based on imagecapturing by the first imaging apparatus is communicable becomes longerthan a time length during which image data based on image capturing bythe second imaging apparatus is communicable.
 6. The control apparatusaccording to claim 5, wherein, in an imaging region to be subjected toimage capturing by the plurality of imaging apparatuses, thepredetermined region is a region to be subjected to generation ofgeometric data for an object based on the plurality of pieces of imagedata.
 7. The control apparatus according to claim 1, wherein the imagingcondition information includes information that enables identificationof a distance between each of the plurality of imaging apparatuses andthe imaging region to be subjected to image capturing by the pluralityof imaging apparatuses.
 8. The control apparatus according to claim 7,wherein, in a case where a distance between a first imaging apparatusincluded in the plurality of imaging apparatuses and the imaging regionis shorter than a distance between a second imaging apparatus includedin the plurality of imaging apparatuses and the imaging region, thedetermination unit determines the time length so that a time lengthduring which image data based on image capturing by the first imagingapparatus is communicable is longer than a time length during whichimage data based on image capturing by the second imaging apparatus iscommunicable.
 9. The control apparatus according to claim 1, wherein theacquisition unit further acquires information indicating a data amountof image data based on image capturing performed by the plurality ofimaging apparatuses in the past, and wherein the determination unitdetermines a time length during which each of the plurality of pieces ofimage data based on image capturing by the plurality of imagingapparatuses is communicable, further based on the data amount indicatedby the information acquired by the acquisition unit.
 10. The controlapparatus according to claim 1, wherein the determination unitdetermines a time length during which each piece of image data iscommunicable during a time period identified based on a frame rate atwhich the plurality of imaging apparatuses performs image capturing. 11.The control apparatus according to claim 1, wherein the determinationunit determines a ratio for dividing a time period identified based on aframe rate at which the plurality of imaging apparatuses performs imagecapturing, based on the information acquired by the acquisition unit,and divides the time period based on the determined ratio to determinethe time length.
 12. The control apparatus according to claim 1, whereinthe determination unit determines a frequency for transmitting a packetof each of the plurality of pieces of image data during a time periodidentified based on a frame rate at which the plurality of imagingapparatuses performs image capturing based on a time lengthcorresponding to each of the plurality of pieces of image data, andwherein the control unit performs control so that communication of apacket of each of the plurality of pieces of image data is performed inaccordance with the frequency determined by the determination unit. 13.The control apparatus according to claim 1, wherein the acquisition unitfurther acquires information indicating a priority of each of theplurality of pieces of image data, and wherein the determination unitdetermines an order to start communication of each of the plurality ofpieces of image data based on the priority indicated by the informationacquired by the acquisition unit.
 14. The control apparatus according toclaim 1, wherein the control unit performs control so that communicationof the plurality of pieces of image data is performed with a data amountcommunicable in the time length determined by the determination unit notbeing exceeded.
 15. The control apparatus according to claim 14,wherein, in a case where a data amount of the plurality of pieces ofimage data exceeds the data amount communicable in the time lengthdetermined by the determination unit, the control unit performs controlso that processing of reducing at least a part of the plurality ofpieces of image data is performed.
 16. A control method that isperformed by a control apparatus, the method comprising: acquiringinformation about an imaging condition of a plurality of imagingapparatuses; determining a time length during which each of a pluralityof pieces of image data based on image capturing by the plurality ofimaging apparatuses is communicable, based on the acquired information;and performing control so that communication of the plurality of piecesof image data is performed in accordance with the determined timelength.
 17. A non-transitory computer-readable storage medium storing aprogram for causing a computer to execute a method, the methodcomprising: acquiring information about an imaging condition of aplurality of imaging apparatuses; determining a time length during whicheach of a plurality of pieces of image data based on image capturing bythe plurality of imaging apparatuses is communicable, based on theacquired information; and performing control so that communication ofthe plurality of pieces of image data is performed in accordance withthe determined time length.